System and method for use of images with recognition analysis

ABSTRACT

An index is provided that holds information about each image content item in a collection of items, For each image content item, a first information item identifying the image content item and its location on a network, and at least one of (i) a second information item identifying a signature value of an object in the image content, or (ii) identification of a recognized object in the image content.

PRIORITY CLAIM/RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.13/088,277, filed Apr. 15, 2011, which is a continuation of U.S. patentapplication Ser. No. 12/431,706, filed on Apr. 28, 2009, which is acontinuation of U.S. patent application Ser. No. 11/543,758, filed onOct. 3, 2006, which (i) claims priority to: U.S. Provisional PatentApplication No. 60/723,349, filed Oct. 3, 2005; (ii) claims priority to:U.S. Provisional Patent Application No. 60/723,356, filed Oct. 3, 2005;(iii) and is a continuation-in-part of U.S. patent application Ser. No.11/246,742, filed on Oct. 7, 2005, which claims priority to U.S.Provisional Patent Application No. 60/679,591, filed May 9, 2005. All ofthe aforementioned priority applications are hereby incorporated byreference in their entirety.

TECHNICAL FIELD

The disclosed embodiments relate generally to the field of digital imageprocessing. More particularly, the disclosed embodiments relate to asystem and method for enabling the use of captured images.

BACKGROUND

Digital photography has become a consumer application of greatsignificance. It has afforded individuals convenience in capturing andsharing digital images. Devices that capture digital images have becomelow-cost, and the ability to send pictures from one location to theother has been one of the driving forces in the drive for more networkbandwidth.

Due to the relative low cost of memory and the availability of devicesand platforms from which digital images can be viewed, the averageconsumer maintains most digital images on computer-readable mediums,such as hard drives, CD-Roms, and flash memory. The use of file foldersare the primary source of organization, although applications have beencreated to aid users in organizing and viewing digital images. Somesearch engines, such as GOOGLE, also enables users to search for images,primarily by matching text-based search input to text metadata orcontent associated with images.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates a system for selecting images for presentation on anonline document in connection with existing content, under an embodimentof the invention.

FIG. 2 illustrates a method to use an index that stores informationabout images, under an embodiment of the invention.

FIG. 3 illustrates a method for supplying a person with pertinentadvertisement, under an embodiment of the invention.

FIG. 4A and FIG. 4B illustrate a result of segmentation on a merchandiseobject, as performed by one or more embodiments described herein.

FIG. 5 illustrates a method for supplying a person with the ability toview merchandise and locate the same or similar merchandise through anon-the-fly image comparison, under an embodiment of the invention.

DETAILED DESCRIPTION

Embodiments provide for a programmatic selection of image content for adocument that is viewed by a person. One or more embodiments provide foran online environment, where the user views a document, and imagecontent corresponding to pictures, advertisements, or other imagecontent is displayed to the user to correspond to content from thedocument that the user views or interacts with. In one embodiment, animage analysis is performed on a collection of image content items toobtain information about each of the items. The analysis results in oneor more of (i) determination of one or more objects in individual imagecontent items, or (ii) determination of a signature value for eachdetermined object. The information obtained from the image analysis in adata structure, such as an index. The data structure is made availablefor search operations that specify one or more criteria determined froma text or image content provided in connection with the document that isviewed by the user.

In another embodiment, an index is provided that holds information abouteach image content item in a collection of items, For each image contentitem, a first information item identifying the image content item andits location on a network, and at least one of (i) a second informationitem identifying a signature value of an object in the image content, or(ii) identification of a recognized object in the image content.

In another embodiment, a computer system is provided for selecting imagecontent item for a document. The system includes a search component thatcommunicates with one or more servers. Each of the one or more serversserve one or more web pages to terminals. The search component isconfigured to be responsive to an input identified from a user of one ofthe terminals interacting with a particular web page to generate acriteria for selecting image content item for a document. In response tothe input containing an image, the search component analyzes the imageto determine one or more objects in the image of the input, and uses theone or more objects determined from the analysis as a basis of thecriteria for selecting the image content.

As used herein, the term “image data” is intended to mean data thatcorresponds to or is based on discrete portions of a captured image. Forexample, with digital images, such as those provided in a JPEG format,the image data may correspond to data or information about pixels thatform the image, or data or information determined from pixels of theimage.

The term signature value means one or more quantitative values thatdistinguish or like an appearance of an object from another object. Thevalues may correspond to vectors or multi-dimensional values. Inaddition, a signature value may be one value, or a collection oraggregate of several other values (e.g. multiple feature vectors).Feature, or feature extraction are other terms that are generally usedelsewhere and have the same meaning that we emphasize with the termsignature.

The terms “recognize”, or “recognition”, or variants thereof, in thecontext of an image or image data (e.g. “recognize an image”) is meantto means that a determination is made as to what the image correlatesto, represents, identifies, means, and/or a context provided by theimage. Recognition does not mean a determination of identity by name,unless stated so expressly, as name identification may require anadditional step of correlation.

As used herein, the terms “programmatic”, “programmatically” orvariations thereof mean through execution of code, programming or otherlogic. A programmatic action may be performed with software, firmware orhardware, and generally without user-intervention, albeit notnecessarily automatically, as the action may be manually triggered.

One or more embodiments described herein may be implemented usingprogrammatic elements, often referred to as modules or components,although other names may be used. Such programmatic elements may includea program, a subroutine, a portion of a program, or a software componentor a hardware component capable of performing one or more stated tasksor functions. As used herein, a module or component, can exist on ahardware component independently of other modules/components or amodule/component can be a shared element or process of othermodules/components, programs or machines. A module or component mayreside on one machine, such as on a client or on a server, or amodule/component may be distributed amongst multiple machines, such ason multiple clients or server machines. Any system described may beimplemented in whole or in part on a server, or as part of a networkservice. Alternatively, a system such as described herein may beimplemented on a local computer or terminal, in whole or in part. Ineither case, implementation of system provided for in this applicationmay require use of memory, processors and network resources (includingdata ports, and signal lines (optical, electrical etc.), unless statedotherwise.

Embodiments described herein generally require the use of computers,including processing and memory resources. For example, systemsdescribed herein may be implemented on a server or network service. Suchservers may connect and be used by users over networks such as theInternet, or by a combination of networks, such as cellular networks andthe Internet. Alternatively, one or more embodiments described hereinmay be implemented locally, in whole or in part, on computing machinessuch as desktops, cellular phones, personal digital assistances orlaptop computers. Thus, memory, processing and network resources may allbe used in connection with the establishment, use or performance of anyembodiment described herein (including with the performance of anymethod or with the implementation of any system).

Furthermore, one or more embodiments described herein may be implementedthrough the use of instructions that are executable by one or moreprocessors. These instructions may be carried on a computer-readablemedium. Machines shown in figures below provide examples of processingresources and computer-readable mediums on which instructions forimplementing embodiments of the invention can be carried and/orexecuted. In particular, the numerous machines shown with embodiments ofthe invention include processor(s) and various forms of memory forholding data and instructions. Examples of computer-readable mediumsinclude permanent memory storage devices, such as hard drives onpersonal computers or servers. Other examples of computer storagemediums include portable storage units, such as CD or DVD units, flashmemory (such as carried on many cell phones and personal digitalassistants (PDAs)), and magnetic memory. Computers, terminals, networkenabled devices (e.g. mobile devices such as cell phones) are allexamples of machines and devices that utilize processors, memory, andinstructions stored on computer-readable mediums.

Overview

FIG. 1 illustrates a system for selecting images for presentation on anonline document in connection with existing content, under an embodimentof the invention. Under one implementation, an image selection system100 such as shown by FIG. 1 may be used to select image content forconcurrent presentation with content that is existing on a web page.Alternatively, the image selection system 100 such as described may beused to replace, append or provide new image content based on a contentthat that the user views or interacts with. As used throughout any ofthe embodiments described herein, the image content that can be selectedand rendered to the user may include any content with images, such assimple pictures (such as provided as JPEG or GIF) or documents or filesthat contain images as a portion (e.g. advertisement media with text andimage).

According to an embodiment, the image selection system 100 of FIG. 1includes a image analysis sub-system 110, an index 120, and a searchcomponent 130. In one embodiment, the image selection system 100 may bemade available as a service to operators of web sites and domains. Tothis end, the image selection system 100 may be made available to one ormore servers 150, each of which interact with terminals 160 operated byusers. The servers 150 provide content in the form of web pages 155 orother online documents or resources. For simplicity, FIG. 1 illustratesone server 150 and one terminal 160, which may be consideredrepresentative of numerous servers or terminals. The image selectionsystem 100 may identify image content that is selected responsively, oron-the-fly, in response to activities of the server 150 or the user inconnection with the downloaded page 155. Under one or more embodiments,such selection may be based on existing content that is eitherdownloaded to the user, detected as being viewed by the user, orotherwise selected or subject to interaction by the user.

One or more aggregation processes 112 or mechanisms may be used toprocure image content in anyone of various forms. The processes 112 maylocate image content items, such as digital images from numerouslibraries, data bases, media collections, or sources for such media thatare accessible over the Internet. Embodiments contemplate the use oflicensed libraries of pictures, and repositories of pictures donated bythe public. In addition, one or more embodiments contemplate use ofimages that, when selected for display by image selection system 100,result in some monetary benefit to a proprietor of that system are ofthe server 150 that rendered image. In the latter case, the images maycorrespond to, for example, advertisements, or images available throughan online service. Such images may be programmatically retrieved, aprovided to a proprietor of system 100 as a library. Numerous othersources of images are contemplated. For example, a web crawler may beused to crawl domains on the Internet and to identify images by filetype (e.g. JPEG). The aggregation processes 112 may store and makepictures available individually for the image analysis sub-system 110.Thus, while specific embodiments described herein may reference use ofimages that can be identified or retrieved through use of the index 120,embodiments provide that the index 120 may associate or referencecontent that incorporates such images. Examples of such content includeadvertisement media (images, text with slogan) or multimedia content.

The image analysis sub-system 110 analyzes individual images as they arelocated from the aggregation processes 112 or otherwise provided to thesystem 100. According to an embodiment, the image analysis sub-system110 performs various kinds of analysis on individual images to identifyinformation about the contents of the images. In one implementation, theimage analysis sub-system 110 outputs the information it determines fromindividual images to the index 120. For each image, the index 120 maystore (i) an identifier 125 to the image and/or to the location of theimage, (ii) information identified from data associated with the image,and (iii) information identified from performing recognition analysis onthe image. Various other kinds of information may be stored. In oneembodiment, information stored with each image includes objectidentification information 127 and a signature value 129 for identifiedobjects. As will be described, the object identification information 127identifies objects in a picture, with a level of specificity orcharacteristic information that may vary amongst implementations. Theobject identification information 127 may be derived from image analysisor text extraction (also described below). The signature value 129, onthe other hand, may be a quantitative expression of an object in theimage. The signature value 129 may be the result of image analysisprocesses.

As mentioned, one of the analysis performed by the image analysissub-system 110 includes extracting text or other information (e.g.metadata) provided with the image to determine information about objectsor content in the image. Image recognition processes may also beperformed to recognize or detect objects from the image. In performingrecognition to identify one or more objects in an image, the imageanalysis sub-system 110 may also identify one or more characteristics ofthe object, such as coloring or salient features. Recognized objects andfeatures of an image include persons, items that appear in the image(e.g. shoes) or even text that appears in the image (such as on a sign).Thus, object identification information 127 may include more than mereidentification of the object. The object identification information mayalso include descriptive information about the object. In oneimplementation, the object identification information 127 is provided astext data to the index 120.

In addition to object identification information 127, the imagerecognition process may be performed to determine the signature value129 of objects detected from individual images. In contrast to objectidentification information 127, the signature value 129 may, under oneimplementation, be a numeric or quantitative description of an object.The signature value 129 can identify one object from another object ofthe same kind. To this extent, the signature value 129 may be used todetermine when two objects are the same (e.g. same person's face, samemodel and type of shoes), or when two objects are similar (e.g. twopeople look alike, or two shoes have similar appearance). The lattercase is referred to as a “similarity comparison”,

Numerous techniques exist to determine objects in images, as well as todetect characteristics of determined objects, and obtaining signaturevalues of objects in images. Some of these techniques are described in,for example, U.S. patent application Ser. No. 11/246,742, entitledSYSTEM AND METHOD FOR ENABLING THE USE OF CAPTURED IMAGES THROUGHRECOGNITION, filed on Oct. 7, 2005; which is hereby incorporated byreference in its entirety. Any of the priority documents may be used intheir teachings for determining objects (including persons, appareletc.) and obtaining signature values for such objects. In addition,sections provided below provide additional information about identifyingobjects from images for specific kinds or categories of objects, andformulating signature values by way of feature vectors.

The following examples are illustrative of the possible information thatcan be outputted by the image analysis sub-system 110 when performingrecognition on individual images (or content containing images). In thecase where an object or feature that is recognized from an imagecorresponds to a face, the recognition may also include, as part of theobject identification information 127, one or more of the following: thecoloring of the person's hair or skin, the gender of the person, theperson's eye color, the race of the individual, and the age group of theindividual. In the case where an object or feature that is recognizedfrom an image corresponds to a shoe, the recognition may also include,as part of the object identification information 129, one or more of thefollowing: the color of shoe; whether the shoe is for a man, woman orchild; the shoe style; the color of the shoe; the shape of the shoe atvarious places, such as the heal or tip; and other salient informationsuch as whether the shoe contains a buckle or zipper, or the pattern ofthe shoe. As mentioned, under one implementation, of the objectidentification information 127 may be in the form of text data. For eachrecognized object in the examples provided, the signature value 129provides a quantitative characterization of the characteristics of thatobject, to distinguish that object from other objects, or like theobject to other object. With the face of a person, this may account forspecific features in the person's face. With objects, such as shoes, thesignature value may quantitatively reflect the shape, color, pattern andsalient features of the shoe.

According to an embodiment, the image analysis sub-system 110 is aprogrammatic component that performs its functions through execution ofcode or logic (i.e. programmatically). One or more alternativeembodiments contemplate the image analysis sub-system 110 as includingmanual operators in addition to programmatic processes. In oneimplementation, object detection is performed by manual operators whoview images individually and categorize or provide information about thecontents of the images. In such an embodiment, the image analysissub-system 110 may provide a human interface to display images from acollection, and accept and record input from operators viewing theimage. Still further, one or more alternative embodiments provide for acombination of manual intervention and programmatic processes to analyzethe contents of the images. In one embodiment, the result ofprogrammatic object detection is displayed to a human operator alongwith the image, and the human operator may verify or correct therecognition. The human operator may also supplement or append therecognition, by, for example, detecting a salient feature that is missedby the programmatic element. In such an embodiment, one implementationprovides for a human operator interface that displays numerous (i.e.tens or hundreds) of images at once, along with the objectidentification information 127, and requires the human operator toverify or correct pictures individually, but as a cluster.

From the output of the image analysis sub-system 110, the index 120 maystore entries that identify or locate individual images, as well as textor quantitative information derived from the image recognitionprocesses. The index 120 may be made available to a search component130, which identifies criteria and uses the information stored in index120 to select images. The search component is responsive to an input 152that is provided from server 150.

Embodiments described herein provided for various kinds of input 152.Input 152 may correspond to one or more of the following: (i) aselection of content that a user on the terminal 160 makes through someaffirmative action, such as selecting and clicking with the mouse orother pointer device; (ii) a detection of content that a user is viewingor is interested in, such as by way of identifying content on a portionof the downloaded web page that the user is viewing; (iii)identification of any content up hearing on the web page; (iv)identification of a subject of a content that the user is viewing.

In order to provide the input 152, server 150 may be configured toinclude a programmatic component that can identify and communicate theinput 152 to the image selection system 100. In the case where input 152corresponds to a selection of content by the user, the user may selectan image, link or other data element that is associated with a subjector topic. The component 158 may communicate the selected topic or linkto the system 100. For example, if the user selects an image or a linkto a movie star, the component 158 may communicate the name of the moviestar to the system 100. Alternatively, the component 158 may identify anassociated topic of the selected item. For example, if the user selectsto view a particular kind of sports car, the component 158 may includeintelligence to identify either the make of the sports car, or just“expensive car”, or another topic that the component 158 is programmedto associate with the selected topic (e.g. a demographic may beidentified by the selection).

In the case where the content on the page 155 is detected, theprogrammatic component 158 may perform similar process to identify, forexample, metadata associated with an image, embedded links in the page,or information pertaining to advertisements that appear on the page. Oneembodiment provides that for text content, programmatic component 158performs a key word search to identify the contents of the document bykeyword. From any of these processes, the programmatic component 158 mayidentify a subject, that is then communicated as part of input 152 tothe system 100. As will be described, what is returned is content,including images, that relate to the input 152.

In addition to implementations and examples provided above, numerousalternatives are also possible. In one embodiment, for example, textinput manually entered by the user forms the basis for determining thesubject for input 152. For example, a search term that the user entersfor a search engine may be used for input 152, or results from thesearch. Alternatively, text the user enters through use of the web page155 can be inspected for keywords, and then communicated as input 152 tosystem 100. For example, the user may enter an email content that isthen inspected for key words, or for its subject line. In either case,identification of the text results in the display of images from system100 that are deemed pertinent in some way to the content that is or wasexisting on the web page 155.

From the input, the search component 130 performs a search using theindex 120. The search component may form a criteria 132 from the input152. One or more embodiments provide that the search component 130 canreceive as input either images or text based data. Accordingly, searchcomponent 130 may include an image analysis component 136 to analyzeimage data as input 152, and text component 138 to analyze text basedinput 152. In one embodiment, the image analysis component 136 formseither a signature criteria or a text criteria. The type of criteria 132and how its implemented to select images depends on implementation, asillustrated by the following usage scenarios. In the case where input152 corresponds to an image, one embodiment provides that the imageanalysis component 136 recognizes or determines an object in the image,and then uses that determination in forming the search criteria 132 onthe index 120. As an alternative or addition, the image analysiscomponent 136 determines the signature value of the object in the image,and uses that value as the criteria (or portion thereof). The signaturevalue may be used when either an exact match to the object in the imageinput is needed, or when a similarity or likeness match is desired tothat object.

For example, the image input may be identified by user input (i.e. theuser selects an image), or provided on a page or portion thereof thatthe user views. From the image input, the image analysis component 136of the search component 130 determines one or more objects in the image.The image analysis component 136 then forms criteria based on thedetermined objects. In one implementation, this criteria may be textbased.

As an alternative or addition, the image analysis component 136determines a signature from the image input. The criteria is then basedon the signature and compared against other image signatures in theindex 120. In one embodiment, a similarity or likeness match may beperformed to identify objects that are similar, or which have similarfeatures. For example, the user may view a merchandise (e.g. rug) forsale, select the image, and direct the image to be compared againstother similar images. The signature of the rug may then be used to forma criteria to perform a similarity match for other rugs, or for othermerchandise carrying a pattern similar to the desired rug.Alternatively, the user may submit image to identify an identicalproduct or design, so that he can compare pricing.

In the case where input 152 corresponds to a text item, the criteria maybe based more directly on the input 152. The text component 138 of thesearch component 130 may translate, parse or otherwise process the inputto form the criteria. Still further, the criteria may be based on bothimage and text, and carry a combination of any of the image analysis ortext component described above.

In either of the cases described, one or more embodiments provide thatthe criteria 132 returns either (i) a set of one or more images, or (ii)identification of the images or their locations. The search componentmay process a result 133 corresponding to the results from the index120. The system 100 may return a set of identification 145 to theselected images, or alternatively, the selected images themselves (withor without other content). The images are then provided in the web page152. Various examples of how images may be provided to supplementcontent, provide advertisement media, or provide merchandise objects.

Alternatively, the system 100 may be provided separate or independentfrom the server 150. For example, a user may simply copy content fromthe web page and visit a domain or site where the system 100 isprovided. In such a case, the input 152 may be provided directly fromthe user, and the output of the identified images may form a new page.

In another embodiment, the index 120 may specify image content that hasnot had image data analysis. For example, index 120 may includeadvertisement media. Information associated with each advertisementmedia may be determined from text associated with the media, or may bemanually determined.

Embodiments described with FIG. 1 may be performed through use of one ormore processors and storage components that are in communication witheach other. In one embodiment, components that form the image selectionsystem 100 may be distributed at different locations on a network, andeven handled by different operators or domains. For example, whileembodiments contemplate that the index 120 is operated in connectionwith the image analysis sub-system 110, one or more alternativeembodiments contemplate that the index 120 and the image sub-system 110are operated independently, at different network locations or sites.

With regard to any of components or elements of a system such asdescribed, one or more embodiments contemplate use of servers,computers, or processing combinations that perform functions such asdescribed with the search component 130 or the image recognitionsub-system 110. Furthermore, the index 120 and other components mayincorporate storage mediums structured in the form of databases or othermemory resources. A system such as described by FIG. 1 may bedistributed over multiple locations on a network such as the Internet,or provided on one domain or even on one server. For example, the index120 and the search component 130 may be provided at different locations,and operated independently of one another by different operators.

Methodology

FIG. 2 illustrates a method to use an index that stores informationabout images, under an embodiment of the invention. A method such asdescribed by FIG. 2 may be performed using a system such as illustratedby FIG. 1. As such, reference to elements of FIG. 1 is intended toillustrate suitable components for performing one or more steps orsub-steps being described.

In a step 210, content that is to be used for image selection isdetected. As mentioned, one embodiment provides that the content isidentified from a third-party server or domain that serves a web page toa terminal. Other embodiments provide that the content is specified bythe user, interacting directly with the image selection system 100.

Step 220 provides that a criteria is determined from the content.Different sub-processes are possible in this step. In one embodiment,the content from step 210 is an image. In the case where the content isan image, two different processes may be operated by the searchcomponent 130. One sub-process that can be performed is a determinationof an object in the image. In step 232, the image analysis component 136of the search component 130 performs analysis to identify objects in theimage. In step 234, a criteria 132 is determined from the determinationof the objects in the image. For example, the image may be analyzed todetermine that there is “rug” or a “patterned rug”. Different levels ofspecificity are contemplated. For example, object determination mayidentify the rug as “Oriental” or “Asian” or predominantly of one color.

Another sub-process that can be performed on the image input is thedetermination of the signature value. In one embodiment, the signaturevalue is determined by detecting or determining the object(s) in theimage in step 236. From image data corresponding to the object, theobject's signature value is determined in step 238. Then in step 240, acriteria may be formed based on the signature value in step 240.

The sub-processes may for image data input may be combined to yield acriteria that identifies the object and its signature. For example, animage of a carpet may yield object determination (“predominantly redrug”) and a signature value (identifying the pattern of the rug).

Alternatively, a text based search criteria 132 may be received from thecontent. For example, text data may be correspond to key words in adocument the person is viewing, or which corresponds to metadata thataccompanies an image. The text data may form the basis of a criteria132.

One criteria 132 may include a combination of all the processesdescribed. For example, a user may view a web page for a carpet forsale. The text for the sale item may be analyzed and parsed to identifythe word “carpet”. From the image, image recognition may yield a “redrug” and the signature value corresponds to a pattern of the rug. Thecriteria 132 may be formed from all three processes, so that it includesspecification of carpet, red rug, and a signature value for the pattern.

Once the criteria 132 is established from the input, step 260 providesthat one or more images are selected for display to the user. Thecriteria 132 may correspond to an output of the image analysis component136 and/or the text component 138,

The index may be scanned to return matching entries in step 270. Theentries may identify the images by location, or alternatively supply theimages directly. The images may be formed in a new web page, or form aportion of an existing web page. Various implementations arecontemplated by which resulting images of the search are returned to theuser.

Advertisement Selection

The following illustrate various usage scenarios that correspond to oneor more embodiments of the invention. Any of the methods described inthe usage scenarios may be implemented using a system such as describedwith FIG. 1.

FIG. 3 illustrates a method for supplying a person with pertinentadvertisement, under an embodiment of the invention. In step 310, aperson is detected as viewing an online content comprising an image.Step 320 identifies the image portion of the content. Step 330 providesthat the image content is analyzed using the image analysis component136. In one embodiment, a step 340 identifies an object of the image. Asan alternative or additional step, step 350 determines a signature ofthe identified object. As another alternative or additional step, step360 identifies text associated with the image. Each of thedeterminations about the image are used to formulate a query to theindex 120, which in the embodiment described, contains advertisementmedia. In step 370, an advertisement media is selected for the userbased at least in part on the image of the content he was viewing.

Merchandise Objects

As mentioned, embodiments described herein may apply to performingobject determination, recognition, and similarity comparison on objectssuch as merchandise. Merchandise objects provide another example of animplementation for a system such as described with FIG. 1.

In determining merchandise from a random image or content, oneembodiment provides that text and metadata information associated withthe image is used as clues to identify the object of the image. In oneembodiment, pre-defined categories are identified, and based oninformation such as keywords describing the image, URL locating theimage, or other information, a categorization of the object in the imageis made. For instance, a website might have named the shoes as “men'sfootwear”. A corresponding pre-defined category may be labeled “men'sshoes”. In one embodiment, a rule-based system can be used to mapdescriptive terms of an image to a predefined category. For instance, ifthe term “shoe”, and “for him” is identified in the descriptive text ofthe image, that item can be assumed to be in men's shoes categories. Therule based system may include a set of rules to perform suchassignments.

In another embodiment, the mappings can be done by a learning algorithm.For such an embodiment, a large collection of data is collected. Alearning algorithm may be trained that can learn automatically thedependency of the words to categories. Optionally, the results of anautomating category mapping algorithm can be verified and updated byhuman operators for accuracy.

In an embodiment, segmentation is performed on the image. Whilecategorization may assist segmentation, segmentation on the image dataitself may be performed independently. The objective of the segmentationprocess is to separate the object(s) of interest from the background.For this, any foreground/background segmentation algorithm can be used.In one embodiment, the background can be assumed to be at the sides ofthe images, whereas the foreground can be assumed to be at the center.The intensity distribution of both foreground and background can beobtained from the center and side pixels respectively. As an example, amixture of Gaussian models can be learnt for the foreground andbackground pixels. As a last step, these models can be applied to thewhole image and each pixel can be classified as foreground andbackground. Optionally, the segmentation outputs of the algorithm can beverified by human operators for accuracy. FIG. 4A and FIG. 4B illustratea result of segmentation on a merchandise object, as performed by one ormore embodiments described herein.

Following segmentation, a process of extraction may be performed. Oncethe object is segmented from the background, features that obtain thecolor, shape, boundary, and pattern of the foreground object arecalculated. Such features may be referred to as “visual features”. Eachof these features may be stored numerically as vectors and each item maybe indexed. For such entries, the index 120 may include in part or wholea similarity database where the item's metadata is saved along with thevisual features. In combination, one by one, or collectively, thevarious feature vectors for an object may comprise the signature value.The item's metadata is also saved as a metadata feature vector. In oneembodiment, the metadata feature can be a mapping of the words to uniqueidentifiers that are derived from a dictionary look-up. Inverse documentfrequency (IDF) of the word can be saved along with this uniqueidentifier. The IDF indicates how frequent the word happens indocuments, and hence how descriptive it is. For instance, if we arelooking at shoe items, the word “shoe” is not very descriptive since ithappens nearly in all the documents (items).

The visual and metadata features can be indexed using various indexingalgorithms. In one embodiment, a linear index can be used where eachitem is stored linearly in a file. In another embodiment, a tree basedindexing algorithm can be used, where the nodes of the tree would keepclusters of similar looking items. This way, only that node needs to beloaded in the search time, and the search may be performed faster.

In one embodiment, once all the items go through the steps of 1)Category mapping, 2) Segmentation 3) feature (signature) extraction and4) indexing, the index database is saved, it is ready to be searched.The search can be initiated from another image. For instance, the userwould tell find more examples (shoes) like another one that they liked.This would make a query to the search index, and a weighted distancematching is applied between the visual and metadata features of thequery image and all the images in the index database. In one embodiment,this weighting can be a linear combination. In another embodiment, thisweighting can be done based on non-linear transformations.

Optionally, once the user gets the results he or she can provideadditional feedback to get more accurate results. Different embodimentscan be based on slider, color picker or based on choosing different keyregions in the image. In the case of sliders, the user is allowed tochange the weights of shape, color or pattern or style of the returneditems. In the case of color picker, the user can choose a particularthat he or she is interested in, and then the algorithm would match thequery image's shape in that particular color. In the case of keyregions, the user draws a rectangle on where he is interested in, andthe system would run a query on local features of that kind. Forinstance, the user can draw a rectangle on the high heel, and thealgorithm looks for high heels on all the images. The results arerefined on run-time, as the users play with any of these feedbackmechanisms.

In one or more embodiments, when index 120 is used to store informationabout images of merchandise objects, the information may include URLs orother links to online merchants that provide the merchandise objects forsale. The URL or link may be returned with images when searches areperformed by, for example, search component 130. Thus, when, forexample, a similarity search is performed, results of the similaritysearch include images that are active, and enable selection by the userto access a site of an online merchant where the merchandise object isprovided.

FIG. 5 illustrates a method for supplying a person with the ability toview merchandise and locate the same or similar merchandise through anon-the-fly image comparison, under an embodiment of the invention. In astep 510, a user selects an image of a merchandise object, such as ashoe or a rug. Step 520 provides that image analysis is performed on theimage of the selected merchandise. From the image analysis, a signaturevalue is determined.

In step 530, a similarity operation may be performed by the searchcomponent 130 on the index 120. The similarity operation may specify themerchandise object and the signature value, or alternatively the variousfeature vectors and other identifying information stored in the index120. In addition, the similarity operation may identify objects that arein images recorded in the index, with signature values that are deemedto be similar to the selected object. In one embodiment, two objects aredeemed similar if the signature values are within a designatedthreshold.

Another technique for performing such a similarity performance is taughtin U.S. patent application Ser. No. 11/246,742, entitled SYSTEM ANDMETHOD FOR ENABLING THE USE OF CAPTURED IMAGES THROUGH RECOGNITION,filed on Oct. 7, 2005; which is incorporated by reference herein. Thus,the similarity comparison may return as a result all images containingan object with a feature vectors and/or signature values that are withina designated threshold that defines the similarity. The threshold may beone of design preference or dictated by the particular application.

In step 540, a result comprising one or more images with objects deemedto be similar in appearance is returned to the user. In one embodiment,the result includes, at least initially, only a single image thatcontains an image deemed most similar to the selected object. In anotherembodiment, a series, sequence or other plurality of images may bedisplayed. The images may be sorted or ranked by various factors,including proximity of similarity or other factors.

In an implementation of an embodiment such as described with FIG. 5, auser may view an auction or e-commerce page that shows an object forsale. The user may select a feature, or alternatively access a site,that accepts the image as input and processes the image to determine asignature value. One or more implementations also provide that textassociated with the merchandise (e.g. auction heading) may also be usedto specify a category that the signature value is to apply to. Then asearch of the index 120 is performed to identify either exact matches(e.g. the same item on sale at another auction or site) or an item thatis deemed similar to the selected merchandise. For example, the user maylike the item being viewed, but may want to see what else that issimilar in appearance is offered at a particular auction site or onother e-commerce sites.

Additional Applications

In one embodiment, an algorithm is used to automatically generate imagesrelated to text content, such as articles. As described in provisionalpatent application No. 60/679,591, tags can be extracted from images,using information obtained from recognition, corresponding to objects inthem, and text in them. Those tags and images are collected in a centralserver. All images are indexed using the tags inside them. In addition,an inverse index is created such that, given a tag, the inverse indexprovides all the images that contain that tag. In addition, a pictureranking algorithm, as described in provisional patent application No.60/679,591 determines the most relevant images with that tag.

In one embodiment, the tagging system can be used for an applicationcalled “photosense”. There are many articles provided with online sourcethat can be supplemented with images. The article might be more valuableand more readable by the addition of relevant photos. As a first step,the system traverses the article, and find the key words. The key wordscan be found by looking at the frequency of words, and looking at thetitle. These words might be filtered by a proper noun dictionary ifnecessary. Once the most relevant words to the article are found, thenthe central server is connected and a search is applied on the relevantwords. The most relevant search image results are returned, and they areautomatically posted next to the images. In addition, an overlay on theimages can be shown when the mouse is on the images. When the userpresses on the overlay, the page might be directed to the web page ofthe actual product item, or full search page of the item from thecentral server. This way, photos are included to add value to thearticle, as well as, ads are displayed in images, and in anon-disturbing manner to the user. This system generates revenues basedon advertisements.

In any one person's library, there are photos of many people. There arephotos of their direct family, photos of their extended family, photosof friends, photos of colleagues, and photos of other people whoattended the same event (wedding, soccer game, etc). Sharing photostoday is labor intensive for the sender and requires tremendous patienceand persistence by the receiver (reminding the other party to pleasesend a copy of that photo). Having a large library of photos that isauto-tagged with who and what is in the photo by itself doesn't solvethe problem. One cannot simply add these photos to a search engine andshow anyone searching all of the photos that match his search, becauseit would violate the privacy of the owner (of the photos). In order tosolve the sharing problem without violating the privacy rights,embodiments contemplate use of a photo sharing system called “gradualphoto discovery”.

As described in Provisional Patent Application No. 60/679,591, first auser adds his photos to the system. The system auto-tags the photo withwho, what, and where the photo is. At this point the photo is markedprivate so no one can see it. A very blurred version of the photo iscreated for both the thumbnail view and the screen resolution view.However the tags (meta-data) about the photo is added to the globalsearch index. Let us assume that this photo has the tags: “Burak”,“Ozge”, “Munjal”, and Las Vegas and is a photo that Burak added. Nowwhen Munjal searches on the photo (assuming he is not aware that Burakis using the system and/or is not linked to Burak as a friend in thesystem) with the keywords Munjal and Las Vegas, the system returns acommunication identifying one or more pictures that match the criteria.However, the returned image would not be shown to the user in itsentirety, but in some degraded, or retracted fashion. For example, theresults may be shown in a very blurred fashion. Munjalcan then select afeature to procure access to the image. For example, an icon or otherfeature may be presented to “Get” access on the blurred photo. Thisenables him to write the submitter (Burak) a message requesting accessto the full image. The submitter may then elect to reply, grant accessor deny access. If he says yes, Munjal can now see the full unblurredphoto. The submitter may also have the option of marking the photo“public” so that anyone can see it from that point forward. In the eventthat Munjal, has multiple search result hits that match his search butare blurred (private) he can request them all at once using a so-called“Get all” button and can write one note to all of the owners.

Traditional photo sharing was always “push” based because of privacyreasons. The addition of meta-data allows to create this new “gradualphoto discovery” system whereby one can first find a photo (using it'smeta-data) and later see the photo (once you get permission using theGet request). Traditionally one could only find a photo once you haveseen the photo not allowing this gradual trust based process to takeplace. An index or database such as shown with various embodiments canfacilitate an embodiment for such gradual image sharing and privacyconcern.

As another example of an embodiment, one or more embodiments contemplateuse of image content items in connection with a matchmaking or datingsite. In one embodiment, a person can specify a celebrity, either byimage or by name. The system 100 may determine the signature values ofimages in a collection corresponding to persons making themselvesavailable on the date site. When the person specifies the celebrity,embodiments identify a signature value of the celebrity or a dress (orshoe) she is wearing. The signature value can be used as part of or thebasis of the search term. The search component 130 can then perform asearch of the index 120 for similar images and return people ormerchandise of a particular type in appearance. As another example, auser can provide images of a person (ex-girlfriend, ex-boyfriend, etc),and the system can return pictures of other similar looking people.Still further, a person may view any random picture from, for example, anews item, and then select the image for a similarity search of othersimilar looking people.

CONCLUSION

As mentioned, it is contemplated for embodiments of the invention toextend to individual elements and concepts described herein,independently of other concepts, ideas or system, as well as forembodiments to include combinations of elements recited anywhere in thisapplication. Although illustrative embodiments of the invention havebeen described in detail herein with reference to the accompanyingdrawings, it is to be understood that the invention is not limited tothose precise embodiments. As such, many modifications and variationswill be apparent to practitioners skilled in this art. Accordingly, itis intended that the scope of the invention be defined by the followingclaims and their equivalents. Furthermore, it is contemplated that aparticular feature described either individually or as part of anembodiment can be combined with other individually described features,or parts of other embodiments, even if the other features andembodiments make no mentioned of the particular feature. This, theabsence of describing combinations should not preclude the inventor fromclaiming rights to such combinations.

What is claimed is:
 1. A method for selecting image content for adocument, the method comprising: performing image analysis on acollection of image content items to obtain information about each imagecontent item, including one or more of (i) determination of one or moreobjects in individual image content item, (ii) determination of asignature value for each determined object; storing the informationobtained from the image analysis in a database; making the databaseavailable for search operations that specify one or more criteriadetermined from a text or image content provided in connection with thedocument.